Web Survey Bibliography
Title Evaluating Three Approaches to Statistically Adjust for Mode Effects
Author Kolenikov, S.; Kennedy, C.
Source Journal of Survey Statistics and Methodology, 2, 2, pp. 126-158
Year 2016
Database Oxford Journals
Access date 19.04.2016
Abstract A major hazard in conducting multimode surveys is the potential for mode effects to compromise the response distributions recorded. We evaluate the strengths and weaknesses of three approaches for statistically adjusting for mode effects. Under a regression modeling approach, adjustments are computed by regressing survey responses on mode, demographics, and other relevant variables. Under a multiple imputation approach, mode effects are conceptualized as a missing-data problem. The standard multiple imputation techniques such as chained equations can be used to impute the responses in additional modes. We also propose a new imputation approach based on an econometric framework of implied utilities in logistic regression modeling. We evaluate all three approaches using data from the second wave of the Portraits of American Life Survey sponsored by Rice University's Kinder Institute for Urban Research. This survey featured online and CATI interviewing with a national sample of adults, with random assignment to either CATI-only or web with CATI follow-up for nonrespondents. We develop a workflow to determine which variables require a mode effects adjustment based on standard false discovery rate multiple hypothesis testing procedures. We detected a significant mode effect on four survey outcomes after controlling for demographics and risk of type I error. The mode effects adjustments were then applied to these variables. The effects on the standard errors and point estimates are examined and discussed along with the advantages and disadvantages of each adjustment approach. The multiple imputation approach produced estimates with better apparent accuracy, as evidenced by better internal consistency of the estimates and a moderate increase in the standard errors. Unlike the regression adjustment approach, which can only produce aggregated estimates for the whole study, the multiple imputation approach can be used for disaggregated analysis with mode-adjusted estimates as well.
Access/Direct link Oxford Journals (Abstract) / (Full text)
Year of publication2014
Bibliographic typeJournal article
Web survey bibliography (183)
- Using experts’ consensus (the Delphi method) to evaluate weighting techniques in web surveys not...; 2017; Toepoel, V.; Emerson, H.
- A Partially Successful Attempt to Integrate a Web-Recruited Cohort into an Address-Based Sample; 2017; Kott, P. S., Farrelly, M., Kamyab, K.
- Overview: Online Surveys; 2017; Vehovar, V.; Lozar Manfreda, K.
- Inferences from Internet Panel Studies and Comparisons with Probability Samples; 2016; Lachan, R.; Boyle, J.; Harding, R.
- Integration of a phone-based household travel survey and a web-based student travel survey; 2016; Verreault, H.; Morency, C.
- Estimation and Adjustment of Self-Selection Bias in Volunteer Panel Web Surveys ; 2016; Niu, Ch.
- Calculating Standard Errors for Nonprobability Samples when Matching to Probability Samples ; 2016; Lee, Ad.; ZuWallack, R. S.
- Establishing the accuracy of online panels for survey research; 2016; Bruggen, E.; van den Brakel, J.; Krosnick, J. A.
- Evaluating Three Approaches to Statistically Adjust for Mode Effects; 2016; Kolenikov, S.; Kennedy, C.
- Linearization Variance Estimators for Mixed ‒ mode Survey Data when Response Indicators are Modeled...; 2016; Demnati, A.
- Options for Fielding and Analyzing Web Surveys; 2016; Schonlau, M.; Couper, M. P.
- Report of the Inquiry into the 2015 British general election opinion polls; 2016; Sturgis, P., Baker, N., Callegaro, M., Fisher, St., Green, J., Jennings, W., Kuha, J., Lauderdale, B...
- Solving the Nonresponse Problem With Sample Matching?; 2016
- Online and Social Media Data As an Imperfect Continuous Panel Survey; 2016; Diaz, F.; Garmon, F.; Hofman, J. K.; Kiciman, E.; Rothschild, D.
- Quota Controls in Survey Research.; 2016; Gittelman, S. H.; Thomas, R. K.; Lavrakas, P. J.; Lange, V.
- Scientific Surveys Based on Incomplete Sampling Frames and High Rates of Nonresponse; 2016; Fahimi, M.; Barlas, F. M.; Thomas, R. K.; Buttermore, N. R.
- Doing Surveys Online ; 2016; Toepoel, V.
- Using Mobile Phones for High-Frequency Data Collection; 2015; Azevedo, J. P.; Ballivian, A.; Durbin, W.
- On Bias Adjustments for Web Surveys; 2015; Fan, L.; Lou, W.; Landsman, V.
- The quality of data collected using online panels: a decade of research ; 2015; Callegaro, M.
- Does the use of mobile devices (tablets and smartphones) affect survey quality and choice behaviour...; 2015; Liebe, U., Glenk, K., Oehlmann, M., Meyerhoff, J.
- Web-based survey, calibration, and economic impact assessment of spending in nature based recreation; 2015; Paudel, K. P., Devkota, N., Gyawali, B.
- Using Web Panels for Official Statistics; 2014; Bethlehem, J.
- Self-reported cheating in web surveys on political knowledge; 2014; Jensen, C., Thomsen, J. P. F.
- Keeping Surveys Valid, Reliable, and Useful: A Tutorial; 2014; Greenberg, M. R., Weiner, M. D.
- Prioritisation of alternatives with analytical hierarchy process plus response latency and web survey...; 2014; Barone, S. Errore, A., Lombardo, A.
- A critical review of studies investigating the quality of data obtained with online panels based on...; 2014; Callegaro, M., Villar, A., Yeager, D. S., Krosnick, J. A.
- Online panel research: History, concepts, applications and a look at the future; 2014; Callegaro, M., Baker, R., Bethlehem, J., Goeritz, A., Krosnick, J. A., Lavrakas, P. J.
- Using Paradata to Predict and to Correct for Panel Attrition in a Web-based Panel Survey; 2014; Rossmann, J., Gummer, T.
- Improving cheater detection in web-based randomized response using client-side paradata; 2014; Dombrowski, K., Becker, C.
- Modelling ”don’t know” responses in rating scales; 2014; Manisera, M., Zuccolotto, P.
- User Modeling via Machine Learning and Rule-Based Reasoning to Understand and Predict Errors in Survey...; 2013; Stuart, L. C.
- Comparison of Three Modes for a Crime Victimization Survey; 2013; Laaksonen, S., Heiskanen, M.
- The Short-term Campaign Panel of the German Longitudinal Election Study 2009. Design, Implementation...; 2013; Steinbrecher, M., Rossmann, J.
- Too Fast, Too Straight, Too Weird: Post Hoc Identification of Meaningless Data in Internet ; 2013; Leiner, D. J.
- Assessing Nonresponse Bias in the Green Technologies and Practices Survey; 2013; Meekins, B., Sverchkov, M., Stang, S.
- Web Panel Representativeness; 2013; Bianchi, A., Biffignandi, S.
- On the Impact of Response Patterns on Survey Estimates from Access Panels; 2013; Enderle, T., Muennich, R., Bruch, C.
- Unit Nonresponse and Weighting Adjustments: A Critical Review; 2013; Brick, J. M.
- Adjusting for bias in a mixed-mode CAWI survey on University students ; 2013; Clerici, R., Giraldo, A.
- A probability-based web panel for the UK: What could it look like?; 2013; Nicolaas, G.
- Panel Attrition: Separating Stayers, Sleepers and Other Types of Drop-Out in an Internet Panel; 2013; Lugtig, P. J.
- Speeding and Non-Differentiation in Web Surveys: Evidence of Correlation and Strategies for Reduction...; 2013; Zhang, Che.
- Web Versus Outbound: A Mode Face-Off Following the Presidential Debate; 2013; Marlar, J.
- The Effects of Errors in Paradata on Weighting Class Adjustments: A Simulation Study; 2013; West, B. T.
- Practical tools for designing and weighting survey samples; 2013; Valliant, R. L., Daver, J. A., Kreuter, F.
- Moving an established survey online – or not?; 2013; Barber, T., Chilvers, D., Kaul, S.
- Measuring working conditions in a volunteer web survey; 2013; de Pedraza, P., Villacampa, A.
- Propensity Score Weighting – Can Personality Adjust for Selectivity?; 2013; Glantz, A., Greszki, R.
- Eurobarometer Special surveys: Special Eurobarometer 381; 2012